Performance image information creating and reproducing apparatus and method

ABSTRACT

Image information representing a manner of playing a piece of music can be diplayed while playing the piece of music. Performance image information is created, which enables displaying an image showing a manner of playing a piece of music while playing the piece of music. A motion component database is created, which comprises a plurality of motion components each created for each musical instrument or each part and stores a trajectory of a fraction of motion of a player and a musical instrument in one of at least one performance method being typical of the each musical instrument or the each part. Motion descriptive information is created, which designates one of the motion components corresponding to performance information on the piece of music to be played and a designated one of the at least one performance method, on a time schedule determined by a predetermined musical time unit of the piece of music to be played. Sequence information is created, which comprises the motion descriptive information, and the performance information. The designated one of the motion components can be edited, and the edited motion component is stored in the motion component database.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to performance image information creatingapparatus and method, and performance image information reproducingapparatus and method, for creating, editing and reproducing performanceinformation and image information including images showing the manner ofplaying a piece of music represented by the performance information, andalso relates to a storage medium storing a program or programs thatimplement these methods.

2. Prior Art

An application program, which is also called “multimedia authoringtool”, is known which is adapted to paste or attach two-dimensionalimages or animation in accordance with a piece of music, using ageneral-purpose computer, such as a personal computer.

The images displayed along with a piece of music according to theapplication program as described above are still pictures or thoserepresented by two-dimensional (2D) image data, but three dimensional(3D) image data cannot be displayed using the above application program.

Further, in editing displayed images, it is not easy to edit such imagesin accordance with playing or performance of a piece of music.

It is also difficult to completely synchronize displayed images withplaying of a piece of music. For example, display of images cannot beeasily changed in synchronism with a change in the tempo of playing of apiece of music, and visual effects cannot be easily applied to images tobe displayed, in synchronism with playing of a piece of music.

SUMMARY OF THE INVENTION

It is therefore a first object of the present invention to provideperformance image information creating apparatus and method, andperformance image information reproducing apparatus and method, whichare capable of displaying image information representing a manner ofplaying a piece of music while playing the piece of music, and toprovide a storage medium storing a program for implementing thesemethods.

It is a second object of the invention to provide performance imageinformation creating apparatus and method, and performance imageinformation reproducing apparatus and method, which are capable ofediting images in accordance with a piece of music, and to provide astorage medium storing a program for implementing these methods.

It is a third object of the invention to provide performance imageinformation creating apparatus and method, and performance imageinformation reproducing apparatus and method, which are capable ofreproducing images or applying visual effects to images that aredisplayed, in synchronism with playing or performance of a piece ofmusic.

In a first aspect of the present invention, there is provided aperformance image information creating apparatus for creatingperformance image information for displaying an image showing a mannerof playing a piece of music while playing the piece of music, comprisinga motion component database comprising a plurality of motion componentseach created for each musical instrument or each part and storing atrajectory of a fraction of motion of a player and a musical instrumentin one of at least one performance method being typical of the eachmusical instrument or the each part, a motion descriptive informationcreating device that creates motion descriptive information thatdesignates one of the motion components corresponding to performanceinformation on the piece of music to be played and a designated one ofthe at least one performance method, on a time schedule determined by apredetermined musical time unit of the piece of music to be played, anda sequence information creating device that creates sequence informationcomprising the motion descriptive information, and the performanceinformation.

Preferably, the predetermined musical time unit is defined by beats ofthe piece of music to be played, or by bars or measures of the piece ofmusic to be played.

Preferably, the motion descriptive information creating device includesan editing device that edits the designated one of the motioncomponents, and stores a result of editing in the motion componentdatabase as an edited motion component.

Preferably, the motion components each comprise recorded information onthe trajectory of the fraction of motion of at least one of each part ofa body of the player and each part of the musical instrument.

Preferably, the motion components each comprise a motion waveformrepresenting the trajectory of the fraction of motion, a sounding pointmarker indicative of timing of tone generation, and a muting pointmarker indicative of timing of tone elimination.

Preferably, the performance image information creating apparatusaccording to the invention further includes a time resolution settingdevice that individually sets time resolutions of waveforms of motion ofthe motion components.

Also preferably, the performance image information creating apparatusaccording to the first aspect of the invention further includes a sceneinformation creating device that creates scene information comprisingscene components including at least background information and lightsource information in the image, the scene components being arranged onthe time schedule, and wherein the sequence information comprises theperformance information, the motion descriptive information, and thescene information.

More preferably, the scene components include information forinstructing information externally entered to be displayed.

In the first aspect of the invention, there are also provided a methodof creating performance image information for displaying an imageshowing a manner of playing a piece of music while playing the piece ofmusic, comprising the steps of creating a motion component databasecomprising a plurality of motion components each created for eachmusical instrument or each part and storing a trajectory of a fractionof motion of a player and a musical instrument in one of at least oneperformance method being typical of the each musical instrument or theeach part, creating motion descriptive information that designates oneof the motion components corresponding to performance information on thepiece of music to be played and a designated one of the at least oneperformance method, on a time schedule determined by a predeterminedmusical time unit of the piece of music to be played, and creatingsequence information comprising the motion descriptive information andthe performance information, and a storage medium storing a program thatis executable by a computer to practice a method of creating performanceimage information for displaying an image showing a manner of playing apiece of music while playing the piece of music, the program comprisinga module for creating a motion component database comprising a pluralityof motion components each created for each musical instrument or eachpart and storing a trajectory of a fraction of motion of a player and amusical instrument in one of at least one performance method beingtypical of the each musical instrument or the each part, a module forcreating motion descriptive information that designates one of themotion components corresponding to performance information on the pieceof music to be played and a designated one of the at least oneperformance method, on a time schedule determined by a predeterminedmusical time unit of the piece of music to be played, and a module forcreating sequence information comprising the motion descriptiveinformation and the performance information.

In a second aspect of the present invention, there is provided aperformance image information creating apparatus for creatingperformance image information for displaying an image in synchronismwith playing of a piece of music while playing the piece of music,comprising an information creating device that creates informationcomprising instruction information arranged on a time scheduledetermined by a predetermined musical time unit of the piece of music tobe played, for instructing image information externally entered to bedisplayed, and a sequence information creating device that createssequence information comprising the information created by theinformation creating device, and performance information on the piece ofmusic to be played.

Preferably, the predetermined musical time unit is defined by beats ofthe piece of music to be played, or by bars or measures of the piece ofmusic to be played.

Preferably, the instruction information includes information forcontrolling visual effects to be applied to the image informationexternally entered.

Preferably, the performance image information creating apparatusaccording to the second aspect of the invention further includes amotion descriptive information creating device that creates motiondescriptive information comprising designation information arranged onthe time schedule, for designating one of a plurality of motioncomponents each storing a trajectory of a fraction of motion of a playerand a musical instrument, the one of the plurality of motion componentscorresponding to the performance information and a performance methodbeing typical of the musical instrument, and wherein the sequenceinformation comprises the information comprising the instructioninformation, and the motion descriptive information.

In the second aspect of the invention, there are also provided a methodof creating performance image information for displaying an image insynchronism with playing of a piece of music while playing the piece ofmusic, comprising the steps of creating information comprisinginstruction information arranged on a time schedule determined by apredetermined musical time unit of the piece of music to be played, forinstructing image information externally entered to be displayed, andcreating sequence information comprising the information created by thestep of creating information, and performance information on the pieceof music to be played, and a storage medium storing a program that isexecutable by a computer to practice a method of creating performanceimage information for displaying an image in synchronism with playing ofa piece of music while playing the piece of music, the programcomprising a module for creating information comprising instructioninformation arranged on a time schedule determined by a predeterminedmusical time unit of the piece of music to be played, for instructingimage information externally entered to be displayed; and a module forcreating sequence information comprising the information created by themodule for creating information, and performance information on thepiece of music to be played.

In a third aspect of the invention, there is provided a performanceimage information reproducing apparatus for generating musical tonescorresponding to performance information and displaying an image showinga manner of playing a piece of music represented by the performanceinformation, based on sequence information comprising the performanceinformation, and motion descriptive information used for displaying theimage showing the manner playing the piece of music, comprising amusical tone generating device that generates the musical tones based onthe performance information, an image creating device that creates theimage showing the manner of playing the piece of music corresponding tothe performance information, based on the motion descriptiveinformation, and a motion component database comprising a plurality ofmotion components each created for each musical instrument or each partand storing a trajectory of a fraction of motion of a player and amusical instrument in one of at least one performance method beingtypical of the each musical instrument or the each part, wherein theimage creating device reads out the motion components from the motioncomponent database, based on the motion descriptive information, andcreates the image using the motion components read from the motioncomponent database.

Preferably, the image creating device creates the image to be displayed,by correcting skeleton data indicative of sizes of the player and themusical instrument contained in each of the motion components, based oninformation contained in the sequence information, for designating asize of the image to be displayed.

Preferably, the performance image information reproducing apparatusaccording to the third aspect of the invention further includes an inputdevice that enters image information from an external device, an imagesynthesizing device that synthesizes an image based on the image createdby the image creating device and an image based on the image informationentered by the input device, and an effects-applying device that appliesvisual effects to at least one of the image created by the imagecorrecting device, the image based on the image information entered bythe input device, and the image synthesized by the image synthesizingdevice, in timing determined by a predetermined time unit of the pieceof music being played, based on the sequence information.

In the third aspect of the invention, there are also provided a methodof reproducing performance image information for generating musicaltones corresponding to performance information and displaying an imageshowing a manner of playing a piece of music represented by theperformance information, based on sequence information comprising theperformance information, and motion descriptive information used fordisplaying the image showing the manner playing the piece of music,comprising the steps of generating the musical tones based on theperformance information, creating the image showing the manner ofplaying the piece of music corresponding to the performance information,based on the motion descriptive information, and creating a motioncomponent database comprising a plurality of motion components eachcreated for each musical instrument or each part and storing atrajectory of a fraction of motion of a player and a musical instrumentin one of at least one performance method being typical of the eachmusical instrument or the each part, wherein the step of creating theimage reads out the motion components from the motion componentdatabase, based on the motion descriptive information, and creates theimage using the motion components read from the motion componentdatabase, and a storage medium storing a program that is executable by acomputer to practice a method of reproducing performance imageinformation for generating musical tones corresponding to performanceinformation and displaying an image showing a manner of playing a pieceof music represented by the performance information, based on sequenceinformation comprising the performance information, and motiondescriptive information used for displaying the image showing the mannerplaying the piece of music, the program comprising a module forgenerating the musical tones based on the performance information, amodule for creating the image showing the manner of playing the piece ofmusic corresponding to the performance information, based on the motiondescriptive information, and a module for creating a motion componentdatabase comprising a plurality of motion components each created foreach musical instrument or each part and storing a trajectory of afraction of motion of a player and a musical instrument in one of atleast one performance method being typical of the each musicalinstrument or the each part, wherein the module for creating the imagereads out the motion components from the motion component database,based on the motion descriptive information, and creates the image usingthe motion components read from the motion component database.

According to the present invention constructed as above, images showingthe manner of playing a piece of music can be displayed in synchronismwith the performance of a piece of music.

Also, the present invention makes it easy to edit images showing themanner of playing a piece of music.

Further, the present invention makes it possible to display actualimages that are obtained by actually shooting an object or objects andinput from external devices, in synchronism with the performance of apiece of music and also apply effects to images that are displayed, insynchronism with the performance of a piece of music, to thereby enhancethe performance effect.

Since the motion components are contained in the database, common motioncomponents can be used with respect to a plurality of performancemethods (performance techniques) and pieces of music, and necessarycomponents can be added to the database, thus assuring an improvedefficiency with which images are created.

Furthermore, a motion component includes a sounding point markerindicative of the timing of tone generation and a muting point markerindicative of the timing of tone elimination, as well as the motionwaveform, thus making it possible to use common motion components uponchanges in the tempo, for example, which results in a reduced size ofthe database.

The above and other objects, features, and advantages of the inventionwill be more apparent from the following detailed description taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a performance image creating apparatusand a performance image reproducing apparatus according to oneembodiment of the present invention;

FIG. 2 is a view useful in explaining a motion component database;

FIG. 3 is a flowchart useful in explaining a process of creating amotion component;

FIGS. 4A to 4C are views useful in explaining creation of motioncomponents in which FIG. 4A shows a manner of acquiring information onthe motion of a player, FIG. 4B shows a case where the motion of amusical instrument is recorded, more specifically, where the swingingmotion of a cymbal is recorded, and FIG. 4C shows a phrase to be played,with three positions indicated therein as representing sounding ortone-generation points, together with respective elapsed times t, t′, t″from the start of performance;

FIGS. 5A and 5B are a flowchart showing a process of creating sequenceinformation, in which FIG. 5A is a flowchart showing the process ofcreating sequence information for each part, and FIG. 5B is a flowchartshowing a process of integrating sequence information of all of theparts and making overall adjustments thereof;

FIG. 6 is a view illustrating one example of image edit screen for eachpart;

FIGS. 7A-7E are views useful in explaining synthesis of a plurality ofmotion components;

FIG. 8 is a view illustrating one example of edit screen on which motioncomponents are edited;

FIG. 9 is a view illustrating one example of image edit screen for eachpart;

FIGS. 10A and 10B are view useful in explaining synthesis of informationon a plurality of parts;

FIG. 11 is a view showing the configuration of a sequence file; and

FIG. 12 is a flowchart showing a sequence information reproducingoperation.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

One preferred embodiment of the present invention will be described indetail with reference to the drawings.

FIG. 1 is a block diagram showing the construction of an apparatus thatoperates as a performance image creating apparatus as well as aperformance image reproducing apparatus as one embodiment of the presentinvention. In FIG. 1, the apparatus includes a central processing unit(CPU) 1 that controls the operations of the whole apparatus, a programstorage device 2 that stores control programs according to which thepresent apparatus is controlled, and a storage device 3 that is composedof ROM, RAM and others to be used as work areas, and stores sequencedata (sequence information) created by means of this apparatus, a motioncomponent database containing components each representing thetrajectory of a performance motion or action in each performance method(performance technique), a scene component database containing scenecomponents, such as background information and light source information,and various other data. The present apparatus further includes akeyboard and a set of operating switches 4 that include variousmanipulable parts provided on an operating panel, and a sound sourceunit 5 that generates musical tone signals to be transmitted through aplurality of channels. The sound source unit 5 may be of any typeselected from, for example, waveform memory type, FM type, physicalmodel type, harmonic synthesis type, formant synthesis type, and analogsynthesizer type of VCO+VCF+VCA. The sound source unit 5 is not limitedto a sound source circuit consisting of a dedicated hardware, but may bea sound source circuit constructed using DSP and a microprogram, or maybe constructed by means of the CPU 1 and a software program. The soundsource unit 5 also includes an effect processing portion that appliesvarious effects, such as vibrato and reverberation, to musical tonesgenerated. The apparatus further includes a sound system 6 that emitsthe musical tones generated from the sound source unit 5.

The apparatus of the present embodiment also includes an image computingdevice (drawing engine) 7 for generating 3D image data and applyingvarious visual effects to images that are displayed, and an imagedisplay device (graphic display) 8 for displaying 3D images generated bythe image computing device 7, and an edit screen that will be describedlater. Alternatively, the CPU 1 may be used to execute image processingwithout using the image computing device 7.

In FIG. 1, reference numeral 9 denotes an external storage device, suchas a hard disc, floppy disc, CD-ROM, MO, and DVD, and 10 denotes a MIDIinterface circuit that enables communications with external MIDIequipment. Reference numeral 15 denotes a communication interface (I/F)that enables sending or receiving of data to or from a server computer102, for example, through a communication network 101.

The apparatus of the present embodiment further includes a videointerface circuit 11 adapted for displaying an image showing the mannerof playing music on a monitor 12 as external equipment connected to theapparatus, an image/picture input device 13 that inputs image or picturesignals from external devices such as television cameras and video taperecorders (VTRs), and a bus 14 that permits data transmission betweenthe components as indicated above.

The hard disc (HDD) mounted in the external storage device 9 may storecontrol programs to be executed by the CPU 1. Where a certain controlprogram is not stored in the ROM of the storage device 3, the hard discmay store the control program, which is read into the RAM of the storagedevice 3 so that the CPU 1 can perform the same operations as in thecase where the control program is stored in the ROM. In this manner, anew control program can be easily added, or an existing program that hasbeen already installed can be easily upgraded.

A control program or various data may be read from a CD-ROM of theexternal storage device 9, and stored to a hard disc mounted in a harddisc drive (HDD). In this manner, a new control program can be easilyinstalled, or an existing program can be easily upgraded.

The communication I/F 15 is connected to the communication network 101,such as LAN (Local Area Network), Internet, or a telephone line, so asto be connected to the server computer 102 via the communication network101. In the case where a certain program or a certain set of parametersis not stored in the hard disc within the HDD, the communication I/F 15may be used for downloading the program or parameters from the servercomputer 102. A client computer (i.e., performance image creatingapparatus and performance image reproducing apparatus in the presentembodiment) sends a command to the server computer 102 via thecommunication I/F 15 and communication network 101, to requestdownloading of a desired program or parameters. Upon receipt of thecommand, the server computer 102 delivers the requested program or setof parameters to the client computer, through the communication network101. The client computer then receives the program or set of parametersthrough the communication I/F 15, and stores it to the hard disc in theHDD. In this manner, downloading is accomplished.

The apparatus of the present embodiment constructed as described aboveincludes a motion component database containing motion components inwhich the trajectories of fractions of the motion of a player or amusical instrument in a typical performance method are recorded, foreach musical instrument or part. The performance image creatingapparatus then reads out motion components corresponding to performanceinformation and a designated performance method, from the motioncomponent database, and creates or edits motion descriptive informationindicative of the manner of playing the piece of music, based on themotion components, so as to generate sequence information containing theperformance information and the motion descriptive information. Also,the performance image reproducing apparatus reads out desired motioncomponents from the motion component database, based on the motiondescriptive information contained in the sequence information, generatesor displays corresponding images, and reproduces or displays thegenerated images or images input from an external device, in synchronismwith the performance of the piece of music.

Motion Component Database

The motion component database will be now described in detail.

The motion component database is created by capturing, for each ofvarious musical instruments or parts, the trajectory of motion of aplayer and a musical instrument in a performance method which is typicalof the musical instrument or part, as motion capture data, resolving themotion trajectory of each part of the body of the player or each part ofthe instrument (hereinafter called “region”), into those of the X, Y andZ-axis directions, and marking the timing of tone generation and toneelimination (beat position in the case of a drum, for example), onto themotion capture data. Each of components of the motion trajectory in theX, Y and Z-axis directions resulting from the resolving of the motiontrajectory is called “motion waveform” and the data representing thefractions of performance motion are called “motion components”.

FIG. 2 shows an example of motion components of a drum part. As wellknown in the art, typical performance methods of the drum include a rolltechnique (single stroke, double stroke), a rim shot technique (closedrim shot, open rim shot), and a technique using a mallet or a brush. Asshown in FIG. 2, each motion component is comprised of data includingmotion waveforms indicative of the trajectory of the motion of theplayer and musical instrument exhibited when a short performance patternis played with each technique, and sounding and muting point markersindicating the timing of tone generation and tone elimination. In thedrum part as illustrated in this example, motion waveforms of aplurality of instruments, such as cymbals, a snare drum and a bass drum,constitute one motion component. In the case of a musical instrument,such as a piano or a saxophone, a motion component is produced for eachinstrument.

A method of creating the motion components will be now described withreference to the flowchart of FIG. 3. Step Sll is initially executed toacquire, as motion capture data, the motion or action of a player andthe motion of a particular musical instrument when the player plays theinstrument in a particular performance technique or method.

FIG. 4A shows the manner of acquiring information on the motion of aplayer. As shown in FIG. 4A, a player, who is equipped with 3Ddigitizers on principal parts (indicated by squares) of his/her body, iscaused to play a particular pattern (a fraction of performance) using aparticular performance technique, and the motion of the body of theplayer during the performance is recorded. The 3D digitizers may bemagnetic or optical ones as known in the art. FIG. 4B shows the casewhere the motion of a musical instrument is recorded, more specifically,where the swinging motion of a cymbal is recorded. In this case, theswinging motion of the cymbal is recorded through motion capturing,using three 3D digitizers mounted on certain positions of the cymbal asindicated by squares in FIG. 4B.

In step S12, the motion capture data obtained in step Sll is processedby resolving the trajectory of the center point of each region on an XYZcoordinate system, to provide a motion waveform indicating the movingstate and position of each region. At the same time, time data may alsobe recorded. In the above example of the cymbal, the motion waveform ofthe region consists of the motion waveform of the center position A ofthe above three points that are regarded as lying on the same plane onwhich A is located. Skeleton size information indicative of dimensionsor sizes of the player and/or the musical instrument on which the motioncapture data are obtained is also recorded.

The control flow then goes to step S13 to store markers (that will-becalled “sounding point markers” and “muting point markers”) thatdetermine coordinates of principal regions at which a tone is generated(sounding point) and a tone is eliminated (muting point), and a periodof time elapsed from a point of time when its performance is started.Where a phrase as shown in FIG. 4C is to be played, for example, threepositions as indicated in the figure represent sounding ortone-generation points, and respective elapsed times t, t′, t″ from thestart of the performance are stored distinguishably. The sounding pointmarkers and muting point markers may be of any type provided that theycan specify the timing of tone generation and tone elimination within aset of motion waveform data (x, y, z, t) acquired as described above.

The control flow then goes to step S14 in which the data acquired in themanner as described above are associated with the performance methodused for giving the performance. At this time, data of closely relatedregions, for example, the right arm, shoulder, elbow, and the knuckle ofeach finger, are stored as combined data of set parts. The operator maychange the combination of the set parts, or resolve data of the setparts into those of individual parts.

In the above manner, data are contained in the database such thatappropriate data can be retrieved in accordance with a change in theposition (change in the shape or size of the player and musicalinstrument) upon reproduction thereof, or a change in the time (ortempo).

For example, the motion components relating to the drum are stored suchthat each component represents the motion of the player or instrumentfrom the time when the player hits each kit to the time when he/shestops hitting the kit. More specifically, a motion component covers themotion of a player raising a stick, hitting a cymbal with the stick, andraising the stick again. Here, the sounding point is defined as thepoint of time when the stick touches the cymbal. The muting point,though it is not necessarily provided, may be defined as the point oftime when the cymbal stops vibrating.

In the case of a piano, a motion component covers the motion of a playerplacing one finger or a combination of fingers on the keyboard, andreleasing the same from the keyboard. In this case, the sounding pointis defined as the point of time when the finger(s) comes into contactwith the keyboard, and the muting point is defined as the point of timewhen the finger(s) is released from the keyboard. A plurality ofvariations of motion components are prepared, from which an appropriateone is selected depending upon the degree of spreading the fingers andthe states of the remaining fingers. For typical chords to be playedwith three or more fingers, a plurality of variations of motioncomponents are prepared from which an appropriate one is selecteddepending upon the manner of using the fingers to play a desired chordor the degree of spreading the fingers.

The motion component may further include data representing the movingvelocity or acceleration of each region, in addition to each data of theXYZ coordinates and time and skeleton size, and the sounding and mutingpoint markers. Further, the motion component may further include datarelating to the shape of the player and/or the shape and texture of themusical instrument.

Creation of Sequence Information

Next a process of creating sequence information (sequence data) asdescribed above will be explained.

FIGS. 5A and 5B are a flowchart showing the process of creating sequencedata. As shown in FIGS. 5A and 5B, the creation of sequence informationroughly consists of a process of creating sequence data for each part,and a process of creating sequence data of all parts by synthesizing orintegrating the created sequence data of respective parts.

FIG. 5A is a flowchart showing the process of creating sequence data foreach part.

Once the process of creating sequence data for each part is started,step S21 is executed to create performance information. This operationis similar to an operation to compose a piece of music using a knownsequence software. With this operation, performance information (MIDIdata) of the part in question is created.

The control flow then goes to step S22 to determine a performance methodwith which the piece of music composed in step S21 is played by the partin question. In step S22, the performance method is determined for eachselected region on a score corresponding to the performance information.If the part in question is a keyboard part, for example, information onchords, fingerwork or fingering, volume and others is entered. In thecase of a drum part, the name of performance method to be used isentered.

Since motion components corresponding to the above information are readout from the motion component database 20 as described above, theoperator is instructed to enter information required for retrievingmotion waveforms of regions from the motion component database 20,depending upon the musical instrument or part selected by the operator.

Once all necessary information is entered, step S23 is then executed toselect and read out motion components corresponding to the performanceinformation and performance method entered in the previous steps, fromthe motion component data 20, create a basic motion waveform to beedited, and generate a 3D image representing the manner of playing thepiece of music based on the basic motion waveform, which image is thendisplayed on a screen. First of all, a player that is initially set inadvance to the musical instrument or part is displayed along with themusical instrument corresponding to tone color information designated inthe performance information of the piece of music.

If the operator is not satisfied with the player or musical instrumentof the basic motion waveform, step S24 is executed to change it toanother player or musical instrument. As to the musical instrument,however, top priority is given to the information relating to themusical instrument that is designated in the performance data of themusic piece. Where the information designates the use of a grand piano,for example, it is possible to change the color of the grand piano fromblack to woodgrain or transparent in graphics, but it is not possible tochange the grand piano into an upright piano or other musical instrumentin a different category. Upon changing of the player or musicalinstrument, available choices of player and musical instrument may bedisplayed on the screen each time the operator wishes to change it, thusassuring improved operating efficiency.

After the player and musical instrument are thus determined, theoperator reproduces the basic motion waveform from the first one to thelast one thereof, and, if he/she is not satisfied with the basicwaveform, editing operations as described below will be performed.

More specifically, the control flow goes to step S25 to perform imageediting operations, such as editing of the motion waveform for eachregion, alteration of coordinates (position) of each region or parts,and alteration of the position of sounding and muting point markers.Also, if the operator could not select a desired player or musicalinstrument in step S24, the control flow goes to step S25 to edit thewaveform by directly reading out appropriate motion components.

FIG. 6 shows an image edit screen, by way of example, that is displayedon the display device 8 when the above steps S23-S25 are executed. Asshown in FIG. 6, the image edit screen includes four windows, i.e., acontrol window 30, an image display window 40, a region selectioninformation window 50, and a parts/region display information window 60.The control window 30 displays control buttons to be operated to playback or rewind a series of images, and the frame position of image datato be edited. The image display window 40 displays a 3D graphic imagerepresenting the manner of playing music. The region selectioninformation window 50 displays respective regions of the player andmusical instrument being displayed, in a hierarchical structure, toenable the operator to select a desired one of the regions. Theparts/region display information window 60 displays the time schedule ofthe motion components for each parts or region.

Here, the control buttons provided in the control window 30 include abutton 31 for returning to the top or initial position of the series ofimages, a button 32 for returning to a previous image one measure beforethe current image, a reverse playback button 33, a stop button 34, aforward playback button 35, a button 36 for forwarding or advancing toan image one measure ahead of the current image, a button 37 forforwarding to the last position of the series of images. Thus, theoperator is able to control reproduction of the 3D images in the imagedisplay window 40 by operating a selected one of these buttons.

In step S23, the player and musical instrument selected by the operatorare displayed in the image display window 40. As initial information fordisplaying the player and musical instrument, the skeleton size data andshape and texture data as indicated above are used. The operator is ableto select a certain region of the image player and musical instrumentdisplayed in the image display window 40, and shift the position of theselected region, to thereby establish a new position of this region. Theselection of the region of the player or musical instrument may also bemade by selecting region display information in the region selectioninformation window 50. Such a region that is located behind anotherregion and is thus difficult to select from the image displayed on thescreen may be selected from the region selection information, so thatthis region can be displayed in the image display window 40 to allowselection thereof. The coordinates of the motion waveform may also beautomatically changed at the same time that the position of the regionis established.

In the region selection information window 50, respective regions thatconstitute the motion components are displayed along with theirhierarchical structure, thus allowing the operator to select one of theregions displayed in this window. In the example illustrated in FIG. 6,respective regions of the drum player and drum set are displayed in ahierarchical arrangement.

The parts display information window 60 includes a section 61 thatdisplays performance information of the part in question, and a section62 that displays the time schedule of each parts.

Once the motion waveforms of the part are created with respect to apiece of music, the time schedule for each parts as a set of motioncomponents is displayed on the basis of, for example, beat as a unittime. In FIG. 6, broken vertical lines in the section 62 representbeats, and solid lines represent bars or measures. The motion waveformof each parts is located at the position where the motion waveform isused within the time schedule of the part, with reference to beats as aunit.

In FIG. 6, reference numeral 63 denotes the time schedule of a partsthat is a drummer in this case, and 64 and 65 denote the motion waveformof each parts relating to the musical instrument. Each motion waveformis displayed in the form of a rectangle, and, if it contains soundingpoint markers, the position of each marker is denoted by a triangle thatpoints to the right. In the example of FIG. 6, the parts 63 includes astick, for example, and the positions of the sounding point markers aredisplayed. Thus, the performance information of the piece of music thathas been already entered is displayed with respect to individual parts,so that the timing of generating and eliminating tones by each parts canbe confirmed on the screen. If the sounding and muting point markers ofthe motion waveform are caused to match the timing of tone generationand tone elimination, musical tones can be generated and eliminated insynchronism with tone-generating (sounding) and tone-eliminating(muting) motions in the performance image.

The playback rate of a motion waveform of each parts can be changed byselecting a rectangle representing the motion waveform of the parts, andchanging the length of the rectangle. For example, the playback rate maybe reduced by increasing the length. The playback rate may also beautomatically changed when the tempo of the piece of music in questionis changed.

Further, a straight line having opposite rectangular ends is displayedwithin a rectangle of each motion waveform, and this straight linerepresents a portion of the motion waveform that is used for actuallyreproducing the image. By changing the length and position of thestraight line, the portion in the motion waveform that is used forreproducing the image may be selected. The length of the straight lineis changed in accordance with changes in the length of the rectangle asindicated above. Also, the lengths of the straight lines and rectanglesare changed on the basis of a predetermined musical time unit such asbeats and bars or measures.

By selecting the rectangle of each motion waveform, each regioncontained in the parts may be selected within the image display window40. This makes it possible to change the motion waveform of each regionto any of other variants or candidates, and edit the motion waveformitself.

It is also possible to change the number of frames used for each region,with the upper limit being defined as the number of frames obtainedthough motion capturing. Thus, the number of frames can be increasedwith respect to regions where subtle movements take place, and reducedwith respect to regions that do not move so much, thus reducing a loador burden on the apparatus upon computing.

Referring next to FIGS. 7A-7E, synthesis of motion waveforms of regionsof a drum part will be explained by way of example. For the drum part,five types of motion waveforms are used which include waveforms of (7A)the motion of the body of a player except fingers, (7B) the motion ofthe fingers, (7C) the motion of hand-held parts or tools (drum sticks)of the instrument, (7D) the motion of stationary parts of theinstrument, and (7E) the motion of movable parts of the instrument. Inthis case, the motion of the player (7A) and the motion of the sticks(7C) are regarded as a set parts, for which a large number of frames areused. On the other hand, the motion of the fingers (7B), the motion ofstationary parts of the instrument (7D), and the motion of movable partsof the instrument (7E) do not require a high resolution, and thereforethe number of frames for these types of motion is reduced as shown inFIGS. 7A-7E.

Furthermore, the motion waveform can be directly edited.

FIG. 8 shows the manner of directly editing the motion waveform. In thisexample, the motion waveform of the region “head (Head A)” is to beedited. If the region “Head A” to be edited is selected in the regionselection information window 50 or the image display window 40, theregion “Head A” and other regions subordinate to the “Head A” (asdenoted by 59 in FIG. 8) which will be influenced by the result ofediting of the “Head A” are indicated in a different form (for example,reversed color) than the other regions in the region selectioninformation window 50. At the same time, the selected region and theregions subordinate to the selected region (namely, the regions that aregoverned or restricted by the selected region) in a 3D image displayedin the image display window 40 are displayed in a different display form(for example, with higher brightness) than the other regions, asindicated by B in FIG. 8. If the operator designates a selected regionby double-clicking, for example, a motion waveform display window 70 asshown in FIG. 8 appears on the screen, and the motion waveform of theregion is displayed in the window 70. The motion waveform shown in FIG.8 is that of Head A, which indicate the angle of a vector directed fromits subordinate region “Sternum” to the “Head A” with respect to theinitial coordinate position of the “Head A”. In the motion waveformdisplay window 70 of FIG. 8, the horizontal axis indicates the number offrames, and the vertical axis indicates the angle, while three linesrepresent angles measured from the X axis, Y axis and the Z axis,respectively.

If the operator changes the position of the region to be edited in theimage display window 40, the motion waveform of this region isautomatically changed. If the motion waveform displayed in the motionwaveform display window 70 is edited, to the contrary, the 3D imagedisplayed in the image display window 40 makes a motion in accordancewith the result of editing of the motion waveform.

In the above manner, the motion waveform can be directly edited. Theediting of the motion waveform is useful when it is desired to smoothlyjoin two or more types of motion that are synthesized or combinedtogether, for example.

After the motion waveform are edited in the above manner, a certain namemay be give to the motion waveform of the region thus edited, and addedto the motion waveform database 20. It is also possible to sequentiallysynthesize or combine two or more types of motion waveform of a certainregion, or divide the motion waveform into two or more types, and newlyadd the results of the synthesis or division to the motion waveformdatabase 20. Once a motion file indicative of the manner of performanceof this part is determined, the name of the motion file, pointers(instrument information, performance method) and other information areautomatically stored in association with performance information.

After sequence data of each part is created in the above manner, theprocess of FIG. 5B of integrating sequence data of all of the parts andmaking overall adjustments thereof is implemented. This process iscarried out using a screen as shown in FIG. 9 for editing data relatingto all of the parts.

Once the process of FIG. 5B is started, step S31 is executed to paste orattach data of each part onto the edit screen. In this step, each partis designated, and data of the designated part is located on the timeschedule along with musical tones and images.

In FIG. 9, reference numeral 80 denotes a window in which the timeschedule of all parts is displayed. The window 80 includes a section 81in which is located scene information relating to a stage, and a section82 that displays data of each part. In the window 80, vertical dottedlines represent beats, and solid lines represent bars or measures, andeach part information is located on the absolute measures and beatsestablished in one piece of music. It is to be noted that the beats andmeasures shown in the edit screen of FIG. 6 and FIG. 8 for editing dataof each part are determined relative to the starting point of the partin question.

The control flow then goes to step S32 to perform an editing operationof scene information. As described above, the scene component databaseis stored in the storage device 3, and the scene component databasecontains various types of scene components used for forming an imagerelating to a performance stage, including background information, lightsource information and camera (point of view) information. In step S32,desired scene components, such as background information, camera (pointof view) information and light source information, are selected from thescene component database, and pasted onto the section 81 of the window80 on the time schedule defined by beats and measures. In the example ofFIG. 9, three items of camera information G0-G2, two items of backgroundinformation G3 and G4 and light source information G5 are pasted on thetime schedule. These scene components are described in, for example,VRML (Virtual Reality Modeling Language).

Further, control information including a image(s) obtained by actuallyshooting an object or objects (hereinafter called “actual image(s)”) andinput from the image/picture input device 13 such as a television cameraor VTR and motion picture files prepared by a computer is also stored asscene components in the scene component data for use as backgroundinformation. The input actual images may be displayed as the backgroundor displayed on a desired display region as a part of the background,for example. Moreover, an object may be shot against a background of asingle color such as blue to obtain an actual image or computer graphics(CG) may be implemented to extract a color or colors other than thebackground from an actual image or a CG image and paste the extractedcolor(s) onto an image or synthesize a CG image and an actual image witha desired transmission factor (chromakey synthesis). In this manner, animage based upon the motion waveform may be synthesized with abackground of an actual image or with a background of a player(s) ofother musical instrument(s) as a scenic element, to thereby display theplayer in question playing together with a CG player on the screen.

Alternatively, an actual image alone may be displayed. For example, anactual image may be inserted in an introduction or an intermezzo so asto enhance the performance effect.

Control information for taking in such actual images, which is also ascene component, is pasted on the time schedule determined by a musicaltime unit such as beats and bars or measures, similarly to other scenecomponents.

Further, effects-applying processes for applying effects to an image maybe adopted as scene components. If such a scene component is pasted onthe time schedule, various visual effects can be applied to an imagedisplayed on the monitor, which is obtained by synthesizing the actualimage and the image generated as above, such as fade-in, fade-out, orwipe at the time of change of the point of view of the image or at thetime of change of the background, changing part of the displayed imageinto a mosaic pattern, and changing the color tone into a sepia tone.The scene components relating to these effects-applying processes may beregistered in the scene component database in the form of functions,making it possible to add desired image processing.

As in the case of the parts as described above, the scene components maybe edited, for example, by selecting a desired one of the scenecomponents on the time schedule of the section 81, and changing thelength of the selected scene component. It is also possible to select acertain scene component or scene information pasted in the section 81,so as to display detailed information of this component and edit anyitem of the detailed information.

A piece of music completed in the above manner may be played back,rewound or forwarded at a certain selected region or regions thereof. Itis also possible to assign a song name to the music piece, and store itwith the name.

Referring next to FIGS. 10A and 10B, there will be explained the casewhere pieces of information on a plurality of parts are synthesized orcombined together. As shown in FIG. 10A, a global stage coordinatesystem is established, in which a personal coordinate system of eachplayer and a coordinate system of a musical instrument played by eachplayer or an object having a certain skeleton structure are respectivelyestablished. The instruments and objects are classified into stationarysystems and partially movable systems, and the origin of the coordinatesystem of each partially movable system is placed on a movable part ofthe system. With this arrangement, the coordinates or positions of theplayers and musical instruments or objects may be freely changed withinthe global coordinate system. In the case of a keyboard of a piano, forexample, coordinate systems are established such that the origin of eachcoordinate system is placed on a movable part of each keyboard, and onlyone type of motion waveform representing the motion of any keyboard isprepared, so that common data (the only one motion waveform) can be usedfor all keyboards by shifting the position of the origin with respect toeach of the keyboards to which the data is applied. With regard tostationary instruments or objects, the initial motion may be maintained.With regard to hand-held instruments or tools, such as drum sticks and aviolin, a local coordinate system is defined within a personalcoordinate system for each player, such that the origin of the localcoordinate system is located on a movable part of the instrument ortool, and the operation of a corresponding motion component isreproduced on the local coordinate system. When the size of a player oran instrument of each part is changed, the ratio is calculated for eachregion such that the skeleton size in the motion component is correctedto a size after the change of the size.

As shown in FIG. 10B, a motion package of each player contains a musicaltone produced by the player and the motion of the player, namely,performance information and motion information, that are packaged insynchronism with each other. The motion packages of respective playersare then positioned with respect to the head position as indicated inFIG. 10B. In this manner, pieces of motion information of each playerand musical instrument are integrated, to thus create sequenceinformation covering all players.

Configuration of Sequence File

FIG. 11 shows the configuration of a sequence file containing thesequence data created in the manner as described above. As shown in FIG.11, the sequence file consists of three files, i.e., (1) a scene file,(2) a personal initial information file, and (3) a personal performancedescriptive file. The scene file (1) contains information relating tothe whole sequence file, more specifically, contains scene information,such as a playback speed or rate, point of view, field of view, lightsource, background information and information instructing taking inimage information from an external device, and information on thepositions of coordinate systems of player and musical instrument of eachpart and sizes thereof. Here, the scene information, such as point ofview and field of view, light source and background information, may bein the form of pointer information for designating relevant scenecomponents contained in the scene component database. The personalinitial information file (2) contains information that determinesskeleton data, shape and texture of each player and musical instrumentfor each part. The personal performance descriptive file (3) contains aperformance information file and a motion descriptive file (motiondescriptive information) with respect to each part. The performanceinformation file is in a MIDI data format (SMF), and the motiondescriptive file contains pointer information that designates motionwaveform data of each player or musical instrument (tool) contained inthe motion component database 20, and the sounding and muting pointmarkers as described above.

Thus, the motion descriptive file included in the sequence file of thepresent invention holds information that permits retrieval of desireddata from the motion component database 20, rather than imageinformation itself. This leads to reduced size and improved portabilityof the sequence file.

Reproduction of Sequence Data

Referring next to the flowchart of FIG. 12, a process of playing orperforming the piece of music in question and displaying an image basedon the sequence data thus created. A similar process to this process isalso executed when a playback button is operated during theabove-described operation to create sequence data.

When the operator determines a piece of music to be played, a sequencefile (reproduction data) of the piece of music is selected from a songdata base 21 that contains a plurality of sequence files correspondingto respective pieces of music that were created as described above. Instep S41, a certain length of the sequence file (reproduction data) ofthe selected piece of music is read out at a time, and the operations ofsteps S42 and S45 are performed based on the data.

In step S46, an operation similar to a conventional automaticperformance operation is performed so as to create tone-generationevents such as key-on events and control changes, and sound sourcecontrol parameters, based on performance information contained in thereproduction data (sequence file) read in step S41. In step S46, thesound source control parameters thus created are transmitted to thesound source unit 5, which in turn generates corresponding musical tones(step S47), and the sound system 6 then emits the musical tones.

In step S42, on the other hand, motion waveform data of each region isread from the motion component database 20, based on the motiondescriptive file contained in the reproduction data (sequence file) readin step S41. The control flow then goes to step S43 to perform acorrecting operation on the coordinate position of the motion waveformof each region read in step S42. At this time, where the operatorselects a part to be displayed on the screen, or change the position ofthe point of view, the coordinate position of the relevant region iscorrected according to the selection or change. More specifically,interpolation is performed on the motion waveform of each region asneeded in accordance with the playback rate of the part, to therebycreate motion data, and spline interpolation is performed on aconnecting part or joint of successive motions. A correcting operationis then performed in which the position of each coordinate system istransferred, rotated or enlarged, for example, depending upon theposition of each region. Also, scene information (stage information) iscreated by reading scene components designated in the scene file, fromthe scene component database 22.

As a result, the model position of the region in question is determined,and animation corresponding to the model position is determined in stepS43.

As described above, each motion component contained in the motioncomponent database 20 includes sounding and muting point markers as wellas coordinate data along the time axis, and the coordinates of eachsounding (tone-generation) point and muting (tone-elimination) point andthe time or velocity from the start of reproduction of the motionwaveform to the sounding or muting point may be obtained based on thesounding point marker and muting point marker. Thus, pictures andmusical tones are generated in synchronism with each other, based on thesounding and muting point markers.

More specifically, when the tempo with which the piece of music isplayed is changed to be k times the tempo (reference tempo) with whichthe motion component is created, the reading frequency of the motionwaveform is reduced (thinning-out) or the same motion position is read aplurality of times, so that the reproduction duration of the motionwaveform is shortened or increased, thus requiring 1/k of the time (or ktimes the velocity) to reach from the start of reproduction of themotion waveform to the tone-generation point. Where the shift time orshift velocity is prepared for each set of coordinates, namely, wherethe motion component contains information on the time or velocity ittakes for each region to move from certain coordinates to the nextcoordinates, the time may be changed (corrected) to be “1/k” of thereference time contained in the motion component, or the velocity may bechanged (corrected) to be “k” times that contained in the motioncomponent.

To simply control the time axis as described above with respect to allmotions may result in unnatural or awkward images. For example, themotion velocity of the whole series of images is reduced to a half ifthe tempo is reduced to a half, and, in the case of performance of adrum, this change in the velocity may result in a series of images inwhich the player touches the drum softly, and the performance looks asif the volume were reduced. To avoid this, positions (from a soundingmotion start point to a muting motion point) relating to a sounding ortone generating operation are established for recognition thereof, andthe motion velocity from the sounding motion start point to the mutingmotion point is not changed even with a change in the tempo.

Also, the motion waveform may be modified by use of sound source controlparameters, such as an envelope or velocity, generated in the abovesound source control parameter generating step S46. Where the motioncomponent in question represents a swinging motion of a cymbal, forexample, the swinging motion of the cymbal may be controlled based onmusical tone control parameters, such as the velocity or track volume.For example, the motion waveform may be created such that the degree ormagnitude of swinging of the cymbal is increased with an increase in thevelocity or track volume, and reduced with a reduction in the velocityor track volume. Thus, a natural manner of playing music can bedisplayed.

The control flow then goes to step S44 in which an image generatingoperation (rendering) is performed on the information determined in stepS43. Namely, a scene is visualized based on the scene information andmotion waveform. More specifically, a 3D animation image or picture iscreated based on the scene information and motion waveform, byperforming operations such as three-dimensional coordinate calculationwith respect to the vertex and normal of each object, illumination andcoloring, texture coordinate calculation, polygon formation, projectioncoordinate calculation, field-of-view clipping, bit map generation, andhidden-surface/transparency processing, and the 3D animation image thuscreated is delivered to a frame buffer.

On this occasion, if the size of the player or musical instrumentactually displayed differs from the size of the player or musicalinstrument previously prepared, the ratio is calculated for each region,to thereby correct the coordinate information and hence correct thedifference in the skeleton size.

Further, if the control signal for taking in images from theimage/picture input device 13 as scene information is pasted on the timeschedule as described above, an actual image or a like image is inputfrom the image/picture input device 13 in timing designated by the sceneinformation, and the input image is synthesized with an image created bythe image generating operation as needed, and delivered to the framebuffer.

Furthermore, if the scene information contains any effects-applyingprocess for applying effects to the image, step S45 is executed to applyvisual effects to the image created in step S44.

In the manner as described above, the manner of playing music or thestate of performance can be displayed in the form of images insynchronism with the performance of music. Also, the output from thedisplay can be stored as motion picture data.

While candidates of motion components are contained in the motioncomponent database 20 such that an appropriate one of the candidates canbe retrieved and generated upon entry of a music piece and theperformance method in the illustrated embodiment, musical notation rulesand symbols may be newly defined that are associated with the respectivemotion components in the database, so that an image can be automaticallyformed at the same time that a score is created using the notation rulesand symbols.

It is to be understood that the object of the present invention may alsobe accomplished by supplying a system or an apparatus with a storagemedium in which a software program that realizes the functions of theillustrated embodiment is recorded, and causing a computer (CPU 1 orMPU) of the system or apparatus to read out and execute the programstored in the storage medium.

In this case, the program itself read from the storage mediumaccomplishes the novel functions of the present invention, and thus thestorage medium storing the program constitutes the present invention.

The storage medium for supplying the above-described program to thesystem or apparatus may be selected from a hard disc mounted in theexternal storage device 9, CD-ROM, MO, MD, DVD, floppy disc, CD-R(CD-Recordable), magnetic tape, non-volatile memory card, ROM, and soforth. The program may also be supplied from the server computer 102 viathe communication network 101.

The functions of the illustrated embodiment may be accomplished not onlyby executing a program read by a computer, but also by causing anoperating system (OS) that operates on the computer, to perform a partor the whole of the actual operations according to instructions of theprogram.

Furthermore, the program read from the storage medium may be writteninto a memory provided in an expanded board inserted in the computer, oran expanded unit connected to the computer, and a cpu, or the like,provided in the expanded board or expanded unit may actually performpart or all of the operations according to the instructions of theprogram, so as to accomplish the functions of the illustratedembodiment.

What is claimed is:
 1. A performance image information creatingapparatus for creating performance image information for displaying animage showing a manner of playing a piece of music while playing thepiece of music, comprising: a motion component database that stores aplurality of motion components each of which corresponds to aperformance method and performance information, each of said motioncomponents representing a trajectory of a fraction of performance motionof both a musical instrument and a player of said musical instrument; aninformation generating-device that generates motion componentinformation designating at least one of said motion components inaccordance with a performance method and performance information on atime schedule determined by a predetermined musical time unit of musicto be played, said performance method representing how to perform saidmusic and said performance information being information for performanceof said music; and a sequence information creating device that createssequence information comprising said motion component information andsaid performance information.
 2. A performance image informationcreating apparatus according to claim 1, wherein said predeterminedmusical time unit is defined by beats of music to be played.
 3. Aperformance image information creating apparatus according to claim 1,wherein said predetermined musical time unit is defined by bars ormeasures of music to be played.
 4. A performance image informationcreating apparatus according to claim 1, wherein said informationgenerating device includes an editing device that edits the designatedone of the motion components, and stores a result of editing in themotion component database as an edited motion component.
 5. Aperformance image information creating apparatus according to claim 1,wherein the motion components each comprise rewarded information on thetrajectory of the fraction of motion of at least one of each part of abody of the player and each part of the musical instrument.
 6. Aperformance image information creating apparatus according to claim 1,wherein the motion components each comprise a motion waveformrepresenting the trajectory of the fraction of motion, a sounding pointmarker indicative of timing of tone generation, and a muting pointmarker indicative of timing of tone elimination.
 7. A performance imageinformation creating apparatus according to claim 1, further including atime resolution setting device that individually sets time resolutionsof waveforms of motion of the motion components.
 8. A performance imageinformation creating apparatus according to claim 1, further including ascene information creating device that creates scene informationcomprising scene components including at least background informationand light source information in said image, said scene components beingarranged on said time schedule, and wherein said sequence informationcomprises said performance information, said motion componentinformation, and said scene information.
 9. A performance imageinformation creating apparatus according to claim 8, wherein said scenecomponents include information for instructing information externallyentered to be displayed.
 10. A method of creating performance imageinformation for displaying an image showing a manner of playing a pieceof music while playing the piece of music, comprising the steps of:creating a motion component database that stores a plurality of motioncomponents each of which corresponds to a performance method andperformance information, each of said motion components representing atrajectory of a fraction of performance motion of both a musicalinstrument and a player of said musical instrument; creating motioncomponent information designating at least one of said motion componentsin accordance with a performance method and performance in formation ona time schedule determined by a predetermined musical time unit of musicto be played, said performance information being information forperformance of said music; and creating sequence information comprisingsaid motion component information and said performance information. 11.A storage medium storing a program that is executable by a computer topractice a method of creating performance image information fordisplaying an image showing a manner of playing a piece of music whileplaying the piece of music, the program comprising: a module forcreating a motion component database that stores a plurality of motioncomponents each of which corresponds to a performance method andperformance information, each of said motion components representing atrajectory of a fraction of performance motion of both a musicalinstrument and a player of said musical instrument; a module forcreating motion component information designating at least one of saidmotion components in accordance with a performance method andperformance in formation on a time schedule determined by apredetermined musical time unit of music to be played, said performanceinformation being information for performance of said music; and amodule for creating sequence information comprising said motioncomponent information and said performance information.
 12. Aperformance image information creating apparatus for creatingperformance image information for displaying an image in synchronismwith playing of a piece of music while playing the piece of music,comprising: an information creating device that creates informationcomprising instruction information arranged on a time scheduledetermined by a predetermined musical time unit of the piece of music tobe played, for instructing image information externally entered to bedisplayed; and a sequence information creating device that createssequence information comprising said information created by saidinformation creating device, and performance information on the piece ofmusic to be played.
 13. A performance image information creatingapparatus according to claim 12, wherein said predetermined musical timeunit is defined by beats of the piece of music to be played.
 14. Aperformance image information creating apparatus according to claim 12,wherein said predetermined musical time unit is defined by bars ormeasures of the piece of music to be played.
 15. A performance imageinformation creating apparatus according to claim 12, wherein saidinstruction information includes information for controlling visualeffects to be applied to said image information externally entered. 16.A performance image information creating apparatus comprising: aninformation creating device that creates information comprisinginstruction information arranged on a time schedule determined by apredetermined musical time unit of the piece of music to be played, forinstructing image information externally entered to be displayed; asequence information creating device that creates sequence informationcomprising said information created by said information creating device,and performance information on the piece of music to be played; and amotion component information creating device that creates motioncomponent information comprising designation information arranged onsaid time schedule, for designating one of a plurality of motioncomponents each storing a trajectory of a fraction of motion of a playerand a musical instrument, said one of the plurality of motion componentscorresponding to said performance information and a performance methodbeing typical of said musical instrument, and wherein said sequenceinformation comprises said information comprising said instructioninformation, and said motion component information.
 17. A method ofcreating performance image information for displaying an image insynchronism with playing of a piece of music while playing the piece ofmusic, comprising the steps of: creating information comprisinginstruction information arranged on a time schedule determined by apredetermined musical time unit of the piece of music to be played, forinstructing image information externally entered to be displayed; andcreating sequence information comprising said information created bysaid step of creating information, and performance information on thepiece of music to be played.
 18. A storage medium storing a program thatis executable by a computer to practice a method of creating performanceimage information for displaying an image in synchronism with playing ofa piece of music while playing the piece of music, the programcomprising: a module for creating information comprising instructioninformation arranged on a time schedule determined by a predeterminedmusical time unit of the piece of music to be played, for instructingimage information externally entered to be displayed; and a module forcreating sequence information comprising said information created bysaid module for creating information, and performance information on thepiece of music to be played.
 19. A performance image informationreproducing apparatus for generating musical tones corresponding toperformance information and displaying an image showing a manner ofplaying a piece of music represented by the performance information,based on sequence information comprising the performance information,and motion component information used for displaying the image showingthe manner playing the piece of music, comprising: a musical tonegenerating device that generates the musical tones based on theperformance information; an image creating device that creates the imageshowing the manner of playing the piece of music con-responding to theperformance information, based on the motion component information; anda motion component database that stores a plurality of motion componentseach of which corresponds to a performance method and performanceinformation, each of said motion components representing a trajectory ofa fraction of performance motion of both a musical instrument and aplayer of said musical instrument; wherein said image creating devicereads out the motion components from said motion component database,based on the motion component information, and creates the image usingthe motion components read from the motion component database.
 20. Aperformance image information reproducing apparatus according to claim19, wherein said image creating device creates the image to bedisplayed, by correcting skeleton data indicative of sizes of the playerand the musical instrument contained in each of the motion components,based on information contained in the sequence information, fordesignating a size of the image to be displayed.
 21. A performance imageinformation reproducing apparatus according to claim 19, furtherincluding: an input device that enters image information from anexternal device; an image synthesizing device that synthesizes an imagebased on the image created by the image creating device and an imagebased on the image information entered by the input device; and aneffects-applying device that applies visual effects to at least one ofthe image created by the image correcting device, the image based on theimage information entered by the input device, and the image synthesizedby said image synthesizing device, in timing determined by apredetermined time unit of the piece of music being played, based on thesequence information.
 22. A method of reproducing performance imageinformation for generating musical tones corresponding to performanceinformation and displaying an image showing a manner of playing a pieceof music represented by the performance information, based on sequenceinformation comprising the performance information, and motion componentinformation used for displaying the image showing the manner playing thepiece of music, comprising the steps of: generating the musical tonesbased on the performance information; creating the image showing themanner of playing the piece of music corresponding to the performanceinformation, based on the motion component information; and creating amotion component database that stores a plurality-of motion componentseach of which corresponds to a performance method and performanceinformation, each of said motion components representing a trajectory ofa fraction of performance motion of both a musical instrument and aplayer of said musical instrument; wherein said step of creating theimage reads out the motion components from said motion componentdatabase, based on the motion component information, and creates theimage using the motion components read from the motion componentdatabase.
 23. A storage medium storing a program that is executable by acomputer to practice a method of reproducing performance imageinformation for generating musical tones corresponding to performanceinformation and displaying an image showing a manner of playing a pieceof music represented by the performance information, based on sequenceinformation comprising the performance information, and motion componentinformation used for displaying the image showing the manner playing thepiece of music, the program comprising: a module for generating themusical tones based on the performance information; a module forcreating the image showing the manner of playing the piece of musiccorresponding to the performance information, based on the motioncomponent information; and a module for creating a motion componentdatabase that stores a plurality-of motion components each of whichcorresponds to a performance method and performance information, each ofsaid motion components representing a trajectory of a fraction ofperformance motion of both a musical instrument and a player of saidmusical instrument; wherein said module for creating the image reads outthe motion components from said motion component database, based on themotion component information, and creates the image using the motioncomponents read from the motion component database.
 24. A performanceimage information creating apparatus for creating performance imageinformation for displaying an image showing a manner of playing a pieceof music while playing the piece of music, comprising: a motioncomponent database that stores a plurality of motion components each ofwhich corresponds to a performance method, each of said motioncomponents representing a trajectory of a fraction of performance motionof both a musical instrument and a player of said musical instrument;and a sequence information creating device that creates sequenceinformation comprising designation information arranged on a timeschedule determined by a musical time unit of performance information,for designating one of the motion components stored in said motioncomponent database, said motion component being designated in accordancewith a performance method determined by said performance information.